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We claim: 



1 1 . A method for detecting one or more objects belonging to the same 

2 object class comprising the steps of: 

3 a), receiving a video sequence comprised of a plurality of image frames; 

4 b). applying one or more classifiers to detect components of objects in an 

5 image frame in the video sequence; 

6 c). computing a confidence score based in part on the response from the 

7 one or more component detectors; 

8 d). repeating steps b). and c). to detect components of objects belonging 

9 to the same object class in additional images frames in the video sequence; and 

10 e). accumulating confidence scores from the component detectors to 

1 1 determine if an object is detected. 

1 2. The method of claim 1 wherein if accumulated confidence scores 

2 indicate high confidence of a presence of an object, the method further 

3 comprising the step of: 

4 identifying the detected components to be an object of a particular type. 
1 3. The method of claim 1 wherein the object class is a vehicle. 

1 4. The method of claim 1 further comprising the step of: 

2 if an object is detected, outputting a detection signal and object position. 

1 5. The method of claim 2 further comprising the steps of: 

2 testing geometry constraints on a spatial arrangement of detected 

3 components in an image; and 

4 applying whole-appearance classifiers an Image patch that contains the 

5 detected components and which is aligned according to the position of the 

6 detected components. 
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1 6. The method of claim 5 wherein the geometry constraints are 

2 derived from camera parameters. 

1 7. The method of claim 5 wherein the geometry constraints are 

2 derived from object size. 

1 8. The method of claim 5 wherein the geometry constraints are 

2 derived from a location of an object appearance in the image. 

1 9. The method of claim 5 wherein the whole appearance classifiers 

2 detect entire or partial object appearance, the entire or partial object appearance 

3 being aligned according to positioning of at least two components. 

1 10. The method of claim 1 wherein the component classifiers include 

2 classifiers for detecting components at multiple scales. 

1 11. The method of claim 1 wherein component classifiers are defined 

^ 2 by discriminant features and decision rules which are learned through boosted 

3 training. 

1 12. The method of claim 1 1 wherein the discriminant features include 

2 corners. 

1 13. The method of claim 1 1 wherein the discriminant features include 

2 horizontal edges. 

1 14. The method of claim 1 1 wherein the discriminant features include 

2 vertical edges. 

1 15. The method of claim 1 1 wherein the discriminant features include 

2 horizontal stripes. 
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1 1 6. The method of claim 1 1 wherein the discriminant features include 

2 vertical stripes. 

1 17. The method of claim 1 1 wherein the discriminant features include 

2 diagonal stripes. 

1 18. The method of claim 1 1 further comprising the step of: 

2 performing an online adaptation to adapt a classifier structure to an online 

3 pattern. 

1 19. The method of claim 18 wherein the step of performing an online 

2 adaptation further comprises the step of: 

3 applying a dynamic switching strategy to direct the detector to take 

4 appropriate weak classifiers as discriminants according to auxiliary information 

5 about the online pattern. 

1 20. The method of claim 2 wherein the one or more classifiers include 

2 overlapping component classifiers. 

1 21 . The method of claim 20 wherein the overlapping component 

2 classifiers comprises four corners representing a rear profile of a vehicle. 

1 22. The method of claim 20 wherein the overlapping component 

2 classifiers comprises four corners representing a frontal profile of a vehicle. 



1 23. The method of claim 20 wherein the overiapping component 

2 classifiers comprises four corners representing a side profile of a vehicle. 

1 24. The method of claim 21 wherein one of the overlapping component 

2 classifiers detects the bottom left corner of a vehicle. 
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1 25. The method of claim 21 wherein one of the overlapping component 

2 classifiers detects the bottom right corner of a vehicle. 

1 26. The method of claim 21 wherein one of the overlapping component 

2 classifiers detects the top left corner of a vehicle. 

1 27. The method of claim 21 wherein one of the overlapping component 

2 classifiers detects the top right corner of a vehicle. 

1 28. The method of claim 21 wherein positioning of the four corners of 

2 the rear profile for a vehicle differ for different classes of vehicles. 

1 29. The method of claim 28 wherein a class of vehicle includes 

2 sedans. 

1 30. The method of claim 28 wherein a class of vehicle includes sports 

2 utility vehicles. 

1 31 . The method of claim 28 wherein a class of vehicle includes vans. 

1 32. The method of claim 28 wherein a class of vehicle includes tractor 

2 trailers. 

1 33. The method of claim 28 wherein a class of vehicle includes trucks. 

1 34. The method of claim 21 wherein a distance between any two 

2 corners of the vehicle is constrained. 

1 35. The method of claim 34 wherein the constraint between any two 

2 corners of the vehicle is scaled based on a distance between the vehicle and a 

3 camera capturing the video sequence and camera parameters. 
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1 36. The method of claim 35 wherein an image pyramid of multiple 

2 resolutions is used to detect objects of size 2*x, 4*x and so on with the classifier 

3 for the size x. 

1 37. The method of claim 1 wherein the accumulated confidence scores 

2 are inferred from confidence scores across multiple frames using a recursive 

3 filter. 

1 38. The method of claim 37 wherein when the accumulated 

2 confidence score is a linear combination of the confidence scores of multiple 

3 component classifiers and the whole-appearance classifiers. 

1 39. The method of claim 38 wherein when the confidence score for a 

2 principal component classifier is sufficiently high, the confidence score of the 

3 remaining component classifiers and the whole-appearance classifier are 

4 computed. 

1 40. The method of claim 2 wherein if an object is detected the method 

2 comprising the step of: 

3 tracking the object over subsequent image frames. 

1 41 . The method of claim 40 wherein the step of tracking the object 

2 further comprises the step of: 

3 restricting an area of search in each subsequent image frame based on 

4 the location of the object in a current image frame. 

1 42. The method of claim 40 wherein the step of tracking the object 

2 further comprises the step of: 

3 determining the optimal classifier scale based on a distance between the 

4 object and a camera detecting the object and camera parameters. 
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1 43. The method of claim 1 wherein the confidence scores of 

2 component classifiers are computed in a coarse to fine framework. 

1 44. The method of claim 1 wherein detection is performed on an 

2 image pyramid of multiple resolutions. 

1 45. The method of claim 1 wherein an object class includes 

2 pedestrians. 

1 46. The method of claim 1 wherein an object class includes bicycles. 

1 47. The method of claim 1 wherein an object class includes 

2 motorcycles. 

1 48. The method of claim 1 wherein object class includes different types 

2 of traffic signs. 
3 

1 49. A system for detection and tracking an object comprising: 

2 a camera for capturing a video sequence comprised of a plurality of image 

3 frames; 

4 a processor for receiving the video sequence and analyzing each image 

5 frame to determine if an object is detected, said processor applying one or more 

6 classifiers to detect components of objects in each image frame and computing a 

7 confidence score based on the response from the one or more component 

8 detectors and the result of additional validation; and 

9 a database for storing the one or more classifiers and object training 
10 samples 
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1 50. The method of claim 49 wherein the object class is a vehicle. 

1 51 . The method of claim 49 wherein the object class is a pedestrian. 

1 52. The method of claim 49 wherein the object class is a bicycle. 

1 53. The method of claim 49 wherein the object class is a motorbike. 

1 54. The method of claim 49 wherein the object class includes different 

2 types of traffic signs. 

1 55. The system of claim 49 wherein the detected components are 

2 determined to be an object if the confidence scores are high. 

1 56. The system of claim 55 wherein if an object is detected, the 

2 processor outputs a warning signal. 

1 57. The system of claim 49 further comprising : 

2 a display for displaying the video sequence. 

1 58. The system of claim 49 wherein the processor further comprises: 

2 means for testing geometry constraints on a spatial arrangement of 

3 detected components in an image; and 

4 means for applying whole-appearance classifiers an image patch that 

5 contains the detected components and which is aligned according to the position 

6 of the detected components. 

1 59. The system of claim 58 wherein the geometry constraints are 

2 derived from camera parameters. 
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1 60. The system of claim 58 wherein the geometry constraints are 

2 derived from object size. 

1 61 . The system of claim 58 wherein the geometry constraints are 

2 derived from a location of an object appearance in the image. 

1 62. The system of claim 58 wherein the whole appearance classifiers 

2 detect entire or partial object appearance, the entire or partial object appearance 

3 being aligned according to positioning of at least two components. 

1 63. The system of claim 49 wherein the component classifiers include 

2 classifiers for detecting components at multiple scales. 

1 64. The system of claim 49 wherein component classifiers are defined 

2 by discriminant features and decision rules which are learned through boosted 

3 training. 

1 65. The system of claim 64 wherein the discriminant features include 

2 corners. 

1 66. The system of claim 64 wherein the discriminant features include 

2 horizontal edges. 

1 67. The system of claim 64 wherein the discriminant features include 

2 vertical edges. 

1 68. The system of claim 64 wherein the discriminant features include 

2 horizontal stripes. 

1 69. The system of claim 64 wherein the discriminant features include 

2 vertical stripes. 
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1 70. The system of claim 64 wherein the discriminant features include 

2 diagonal stripes. 

1 71 . The system of claim 64 further comprising the step of: 

2 performing an online adaptation to adapt a classifier structure to an online 

3 pattern. 

1 72. The system of claim 71 wherein the step of performing an online 

2 adaptation further comprises the step of: 

3 applying a dynamic switching strategy to direct the detector to take 

4 appropriate weak classifiers as discriminants according to auxiliary information 

5 about the online pattern. 

1 73. The system of claim 49 wherein the one or more classifiers include 

2 overlapping component classifiers. 

1 74. The system of claim 73 wherein the overlapping component 

2 classifiers comprises four corners representing a rear profile of a vehicle. 

1 75. The system of claim 74 wherein one of the overlapping component 

2 classifiers detects the bottom left corner of a vehicle. 

1 76. The system of claim 74 wherein one of the overlapping component 

2 classifiers detects the bottom right corner of a vehicle. 

1 77. The system of claim 74 wherein one of the overlapping component 

2 classifiers detects the top left corner of a vehicle. 

1 78. The system of claim 74 wherein one of the overlapping component 

2 classifiers detects the top right corner of a vehicle. 
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1 79. The system of claim 74 wherein positioning of the four corners of 

2 the rear profile for a vehicle differ for different classes of vehicles. 

1 80. The system of claim 79 wherein a class of vehicle includes sedans. 

1 81 . The system of claim 79 wherein a class of vehicle Includes sports 

2 utility vehicles. 

1 82. The system of claim 79 wherein a class of vehicle includes vans. 

1 83. The system of claim 79 wherein a class of vehicle includes tractor 

2 trailers. 

1 84. The system of claim 74 wherein a distance between any two 

2 corners of the vehicle is constrained. 

1 85. The system of claim 84 wherein the constraint between any two 

2 corners of the vehicle is scaled based on a distance between the vehicle and a 

3 camera capturing the video sequence as well as camera parameters. 

1 86. The system of claim 85 wherein an image pyramid of multiple 

2 resolutions is used to detect objects of size 2*x, 4*x and so on with the classifier 

3 for the size x. 

1 87. The system of claim 49 wherein the accumulated confidence 

2 scores is inferred from confidence scores across multiple frames using a 

3 recursive filter. 

1 88. The system of claim 87 wherein when the accumulated confidence 

2 score is a linear combination of the confidence scores of multiple component 

3 classifiers and the whole-appearance classifiers. 
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1 89. The system of claim 87 wherein when the confidence score for a 

2 principal component classifier is sufficiently high, the confidence score of the 

3 remaining component classifiers and the whole-appearance classifier are 

4 computed, 

1 90. The system of claim 49 wherein the processor comprises: 

2 means for tracking a detected object over subsequent image frames. 

1 91 . The system of claim 90 wherein tracking means further comprises: 

2 means for restricting an area of search in each subsequent image fi^ame 

3 based on the location of the object in a current image frame. 

1 92. The system of claim 90 wherein the tracking means further 

2 comprises: 

3 means for determining the optimal classifier scale based on a distance 

4 between the object and a camera.detecting the object and camera parameters. 

93. The system of claim 49 wherein the confidence scores of 
component classifiers are computed in a coarse to fine framework 

1 94. The system of claim 49 wherein detection and tracking is 

2 performed on an image pyramid of multiple resolutions. 

1 95. A method for tracking one or more objects depicted in a video 

2 sequence comprising: 

3 receiving a video sequence comprised of a plurality of image frames; 

4 detecting an object to be tracked in one of the image frames; 

5 computing one or more appearance trajectories based on the position of 

6 the tracked object to estimate the direction in which the tracked object is 

7 traveling; and 
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8 determining if the object is the tracked object based on the appearance 

9 trajectory. 

1 96. The method of claim 95 wherein said step of determining if the 

2 object is the tracked object further comprises the steps of: 

3 applying one or more classifiers to the computed appearance trajectory; 

4 computing a confidence score for the computed appearance trajectory; 

5 and 

6 determining the object to be the tracked object if the confidence score is 

7 high. 

1 97. The method of claim 95 wherein the computation of the 

2 appearance trajectory considers physically well-founded motion constraints. 

1 98. The method of claim 95 wherein the trajectories are computed by 

2 accumulating detection results over multiple frames. 

1 99. The method of claim 98 wherein the computed appearance 

2 trajectory determines changes in position, scale, location and aspect conditions 

3 in consecutive frames. 

1 1 00. The method of claim 95 wherein the tracked object is a vehicle. 

1 101. The method of claim 95 wherein the tracked object is a pedestrian. 

1 1 02. The method of claim 95 wherein the tracked object is a bicycle. 

1 103. The method of claim 95 wherein the tracked object is a motorbike. 

1 104. The method of claim 95 wherein the tracked object includes 

2 different types of traffic signs. 



